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ABSTRACT 


Artificial intelligence (“intelligent agent”) teehnology is widely deployed in 
numerous eommereial areas sueh as networking, modeling, and software; however, this 
teehnology remains under-utilized by operational organizations within the United States 
Navy. This thesis will investigate the importanee of artifieial intelligenee (AI) for 
military training simulations, partieularly in the training of intelligenee personnel in the 
Navy. 

The Computer Generated Forees (CGF) of the eurrent Intelligenee Team Trainer’s 
(ITT) system initiate aetions as a result of rigid seripted programming. Forees will 
exeeute the same aetions regardless of what the user deeides to do, resulting in highly 
unrealistie seenarios. For instanee, in a seenario where an ARG (Amphibious Ready 
Group) transits the Strait of Hormuz, the response of Iranian P3 or an ineoming dhow 
would be the same whether the battle group utilized frigate eseorts or not. 

This thesis will produee very simple, but less rigid AI, whieh ean easily be made 
more eomplex and “intelligent” in later phases. Demonstrations and assessments will 
validate the importanee of AI integration for the ITT. Furthermore, this analysis of the 
requirements for the AI will assist training eommands and eombat information eenters 
fleet wide with the range of realistie eombat-related possibilities needed in order to 
ensure a fully eapable ‘eombat ready’ wateh team. 
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I. 


INTRODUCTION 


A. PROBLEM 

In the Department’s Transformation Planning Guidanee, Seeretary of Defense 
Rumsfeld stated, “We must transform not only the eapabilities at our disposal, but also 
the way we think, the way we train, the way we exercise, and the way we fight.” 
(Training04) Within this document we focus on how ‘training’ within the military and 
other internal components within the DoD must now prepare the force to learn, 
improvise, and adapt to constantly changing threats while simultaneously executing 
doctrine to standards. While it is obvious that training of some kind is essential in any 
strategic environment, the method of approach is just as vital. That is where the use of 
computer simulation training plays a major role in the military today. More and more 
computer-based training is being introduced into the military as a means to not only save 
money, but to allow capabilities that would otherwise be limited to the real-world and 
access to more than one individual at a time. 

Despite many advances achieved within the fields of Modeling and Simulation 
(M & S) and agent technology over the past several decades, practical application of both 
have been highly under utilized by the operational units of the United States Navy. When 
such technology has been deployed with units, it has usually been developed in order to 
improve operator proficiency, which is also the focus of this work. Little has been 
created for the war fighter to run “what-if ’ scenarios that might aid in development of 
tactical planning for increased proficiency of established doctrine. 

B. MOTIVATION 

Currently, battle group (BG) intelligence teams are required to spend a minimum 
of three training sessions at the Intelligence Team Trainer (ITT) at the Fleet Intelligence 
Training Center, Pacific (FITCPAC). This trainer is a highly realistic simulation of the 
various intelligence cells onboard ships and trains the teams within the different cells as 
well as coordination between the multiple cells of the BG. Resembling many trainers 
currently employed fleet wide, the trainer is not meant to train individuals about their 
jobs, for job competency is trained earlier and is assumed by the time the team reaches 
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FITCPAC. The trainer is utilized as a way to teach the team to function proficiently and 
that a team’s value is the sum of all of its parts. 


Within the ITT, operators sit at mock consoles that replicate actual seating 
arrangements that are utilized on deployment and all input into these systems are highly 
accurate. Accurate message traffic, including some that do not relate to the problem, is 
distributed though out the cells. See Figure 1 below. 



Figure 1. SUPPLOT Team Trainer Stations (From: MOVES Institute Proposal) 


As intelligence analysts, the team members interpret and use this information to 
make a determination on the operation situation that is then passed on to BG commanders 
via their Watch Officer. Due to the substance of the scenarios, the details of each are 
tightly scripted and take a significant amount of time to design and build. 

The amount of time it takes to create each scenario is one of many obstacles to the 
ITT. Currently, each scenario takes up to one year to build and as a result few scenarios 
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exist which reflect the latest conditions. This has resulted in each BG intelligence team 
conducting the same scenario multiple times. Without unpredictability and variation, the 
team’s performance becomes less that of an ability of a team to function properly, but 
rather their ability to remember each scenario. Additionally, there is no way for the BG 
intelligence team’s to train on scenarios based on the latest changes in the tactical, 
operational, or strategic situation. 

C. THESIS STATEMENT 

This thesis describes the investigation, research, and development of a less rigid 
artificial intelligence (AI) which can easily be made more complex and “intelligent” in 
later phases. Demonstrations and assessments validate the importance of AI integration 
for the ITT. Furthermore, this analysis of the requirements for the AI will assist training 
commands and combat information centers fleet wide with the range of realistic combat- 
related possibilities needed in order to ensure a fully capable ‘combat ready’ watch team. 
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II. BACKGROUND INFORMATION 


A. INTRODUCTION 

Due to the requirements of their work, intelligence professionals are not always 
given the opportunity to train as a team prior to a deployment. A response to this need 
was developing an Intelligence Team Trainer supervised by military instructors and 
civilian contractors that allowed these teams to not only meet fleet requirements for 
inspections, but to allow them to train in a manner similar to their deployed environment. 
The trainer establishes rigorous skill sets that build upon one another, so the end result is 
an effective and efficient intelligence operation that can easily communicate with other 
intelligence operations and teams throughout not only their own Battle Group, but 
through out the world’s intelligence cohorts. 

The intended end state of each training simulation is to expose the teams to 
situations that they will encounter while underway. In other words, the trainer’s intention 
is to better prepare a sailor, allowing them have a better grasp on what to do and how to 
react in a timely manner for various events they will encounter in the real world. 
Additionally, a more general goal of this trainer is to give the Sailors the opportunity to 
learn how to work together as a team in an Operational setting. 

B. GLOBAL COMMAND AND CONTROL SYSTEM (GCCS) 

The Global Command and Control System (GCCS) is an automated information 
system designed to support a common operation picture with the use of an integrated set 
of analytic tools and the flexible data transfer capabilities. GCCS is fast becoming the 
single C4I system to support the warfighter from foxhole to unified command posts 
(GCCSOO). 

The purpose of GCCS is to provide command centers with C5I, maintain the 
common operating picture (COP) for the specified area of responsibility (AOR), and 
provide the DoD with an open system architecture, as shown in Figure 2. 
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Figure 2. Illustration of GCCS Elements (From: 
http://www.fas .org/nuke/guide/usa/c3i/gccs 1 .gif) 


GCCS is known to most in the military as the god’s eye view of the world. It 
enables trained personnel to keep track of units (US, friendly, and hostile) throughout the 
world on one screen with tools that allow for modifications and transfering data to and 
from databases maintained by the Defense Intelligence Agency (DIA). 


There are four capabilities and operations of GCCS: 

1. Databases 

• National, Tactical, and Local data 

• Intelligence Databases (Modernized Integrated Database (MIDB), 
populations, units, facilities) 


2. Imagery 

• If stored, can be manipulated 

• JIVE is the MIDB and image link program (PC side) 
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• Good to use C2PC or Intel Office to maintain 

3. Message Handling and Communications Processing 

• Lines of Communication (LOC’s) and IR’s 

4. Track Correlation for near real-time update 

• For Force Over-the-Horizon Track Coordinator (FOTC) 

• Locator data 

• Longitude and Latitude 

• Link data (11/14/16) 

In order to operate this system deployed or land-based, manning of GCCS 
involves two main categories of personnel: users and maintainers. 

Users 

1. Intelligence Specialist (IS) - Tactical / Strategic Intelligence 

2. Operations Specialist (OS) - FOTC 

3. Electronic Warfare (EW) - Maintain radars 

4. Cryptologic Technician (CT) - EEINT correlation and database 

Maintainers 

1. Electronic Technicians (ET) - Maintain hardware 

2. Information Systems Technician (IT)/ Cryptologic Technician - 
Communications (CTO) - software and application 

The users and maintainers of the system are Navy enlisted personnel with 

sub specialties that qualify them to train for this system. In general, each rate has 

different positions within the trainer and requires a prerequisite such as a special school 

or qualification in order to do their job. Eor instance, the Intelligence Specialist (IS), if 

attached to a ship, must attend the Global Command and Control System Maritime Eleet- 

School (GCCS-M E) held at the EITCPAC center in San Diego (for West Coast 

personnel). Once they graduate from the school they receive a certification that qualifies 

them to operate the GCCS system and any application related to the system. 

Additionally, each piece of equipment or application in the system requires a certification 

from its associated training environment. 

GCCS is the principal command and control system supporting the Joint Chiefs of 
Staff (JCS) and Commanders in Chief (CINCs) in managing military assets. GCCS 
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support six mission areas (operations, mobilization, deployment, employment, 
sustainment, and intelligence) through eight functional areas: 

• Threat identification and assessment 

• Strategy planning aid 

• Course of action development 

• Execution planning 

• Implementation 

• Monitoring 

• Risk analysis 

• Common tactical picture 

C. MODERNIZED INTEGRATED DATABASE (MIDB) 

The Modernized Integrated Database (MIDB) is considered the worldwide 
general military intelligence (GMI) database for the Distributed Production Program 
(DPP) to provide GMI intelligence to the warfighter. MIDB is currently available 
through Intelink, the Sensitive Compartmentalized Information network. MIDB provides 
access in several different forms such as static posting, templated dynamic pull, and full 
MIDB access via Telnet. MIDB serves as the principal database of intelligence data for 
the entire United States Department of Defense (DOD) community, as well as Australia, 
Canada, and the United Kingdom. MIDB Order of Battle data contains all official 
validated graphical and textual information about enemy sites, facilities, and units. The 
system supports MIIDS/IDB database (military forces, installation and facilities, 
population concentrations, command and control structures, significant events and 
equipment) for retrieval and maintenance. MIDB is the MUDS database with the 
addition of icons that represent the data in the database, as shown in Figure 3. Data 
collected from this database is extremely important to the intelligence team trainer and is 
pulled on a daily basis to maintain an updated operational picture of the world and, more 
importantly, the area of operation. 
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Figure 3. Mainframe IDB (From: 
http://www.fas.org/irp/program/disseminate/midb_build.gif) 


MIDB is a DOD migration system and currently includes several legacy systems: 

• Electronic Order of Battle Services (EOBS) 

• Expeditionary Warfare 

• Military Eacilities Eile (MIEEAC) 

• Target Material Management (TMM) 

• CENTCOM/SOCOM Integrated Data System (CSIDS) 

• Eorce Trends database (EORT) 

• Eorce Tracking Information System (EORTRIS) 

• Space Data Base (SDB) 

D. ARTIFICIAL INTELLIGENCE 

There has been a revolution of change in game design, slowly evolving from a 
linear aspect, such as scripted game play to an environment of exponential 
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unpredictability (TozourOS). These changes have brought about numerous approaches 
and techniques relevant to applying variability to responding to a user’s actions in order 
to create a more realistic training environment. Among these techniques are genetic and 
heuristic algorithms, scripting, state machines. A* Path finding, and Bayesian networks. 
According to Paul Tozour, the most useful of these in creating a more human-like 
reasoning is the probabilistic inference technique of Bayesian networks (Touzour02_l, 
Touzour02_2). Additional contributions have been made by the entertainment sector, 
and due to advancements in hardware and software continuously become more accessible 
to the everyday user. 

For decades the military has built computerized tools for analyzing combat such 
as, ATLAS, CASTFOREM, THUNDER, and TACWAR, to name a few. Recently, 
simulations have taken over as the foundation for computerized combat training tools 
(Janus, AWSIM, and JTES). The expense of producing these computerized training tools 
is phenomenal. Realizing the cost involved in production and the continuing influence of 
producing accurate training systems, the Department of Defense (DoD) made the 
validation, verification, and accreditation (VV&A) of these simulations an official 
requirement (DOD5000, Training04). 

The difficulty in conducting VV&A for a particular system is daunting and 
requires individuals with a great deal of experience in building and utilizing military 
simulations. Numerous papers have been written in this area; however, little is written on 
the effects of systems with rigid programming (i.e., scripted events) versus systems 
utilizing AT In an article written by John E. Eaird, he studies the similarities and 
differences in AI for computer games and AI utilized in computer generated forces. 
(EairdOO) 

In most legacy military training tools, the framework behind the system exploits 
scripted events. In other words, enemy actions behave the same every time the 
simulation is run. When a validation of the effects of these types of systems on trainee 
performance is conducted over a sustained period of time, performance levels of 
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individuals/teams continually increase. Of course, it is painfully obvious why: the 
individual/team knows exactly what the enemy will do each time. But this cannot be 
called effective training. 

Many examples of practical approaches to applying AI to game-based training are 
available to the military. Among them is an article by John Manslow who discusses the 
importance of the application of using AI in order to force players to continually search 
for new ways to defeat the enemy AI. He provides solutions and steps in the decision 
processes of making an adaptive AI. (Manslow02). 

The objective of many training systems is enhanced training value with no 
negative training. Detail or the ‘fidelity’ of a training system is dependent on the goal of 
the trainer. There are numerous papers that discuss training transfer or increasing or 
decreasing fidelity based on the objective of the simulation or virtual environment. For 
this reason, this paper will focus on performance measures and explaining how to obtain 
them in a system that utilizes very simple AI. The metric obtained in this analysis will 
demonstrate the superiority of using simple AI rather than none at all in combat training 
tools. 
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III. REQUIREMENTS ANALYSIS 


A. EVALUATING TRAINING GOALS 

During the researcher’s visit to the Intelligence Team Trainer (ITT), instructors 
discussed their desired functionality in a new training system. The instructors expressed 
great interest in the use of AI in order to allow them to utilize time spent in the trainer to 
overlook the team itself, instead of typing message traffic or equivalent activities that 
take away from complete observation of the team. Additionally, the instructors desired 
pull-down menus to set-up various scenarios based on team experience and time 
limitations. The pull-down menus include a range of selections from areas of 
responsibility (AORs), time of day, weather, countries involved, etc. This will be 
covered in more detail in the Design Chapter. 

In a meeting conducting between FITCPAC personnel and the MOVES Institute 
at Naval Postgraduate School, the following critical factors for a successful game-based 
simulation were discussed. The trainer should have all the basic functionality of the old 
system in addition to the following: 

• The trainer should be simple enough to use so that both staff at FITCPAC 
and deployed personnel can utilize the trainer and produce scenarios 
within without special training; 

• The trainer should be PC based in order for personnel to use the 
simulation exclusive of any access to specialized systems which might be 
occupied by a team or inoperative do to scenario setup. The specialized 
systems would include the MIDB, GCCS, and Office Intel; 

• The system should be capable of training a complete intelligence team, 
requiring a network that allows team members to interface with one 
another; 

• The system should be capable of training a single operator, with the other 
team members simulated by agents that are indistinguishable from actual 
humans. For instance, if one user desires to operate a PC based version of 
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the trainer, the individual stations not manned will be simulated by the 
game with responses and actions expected from an actual qualified watch 
stander; 


• The trainer should be capable of interfacing and stimulating the systems 
and equipment involved in normal use by the intelligence team in order to 
create the same environment that is takes place while deployed; 

• The trainer should be reactive, in order that the user’s actions produce the 
enemy’s actions. The system will also permit the enemy to “learn” from 
the user’s actions so that the same user action will not produce identical 
results each time; 

• The trainer should be entertaining and fun, so that Sailors will play the 
simulation outside of the work environment allowing for maximization of 
training. 

B. THESIS SCOPE LIMITATIONS 

Due to time and capability limitations, this thesis provides the basic functionality 
of the current system’s capabilities limiting itself to one scenario. Additionally, this 
thesis provides a simple PC based version of the trainer with a scenario generator, simple 
MIDB, message traffic implementation, and a graphical user interface that displays the 
actions of the user and the enemy’s response to that action. This thesis focuses only on 
the issue of the reactivity of the trainer to user actions. However, one scenario will be 
sufficient enough to provide evidence whether utilizing artificial intelligence is an 
improvement over the older scripted version. 

C. CURRENT SYSTEM FUNCTIONALITY 

The ITT utilizes the framework from GCCS, MIDB, Office Intel, and GCCS is 
the main operating system for the ITT simulation at FITCPAC, San Diego. Use of a 
modified stand-alone version is maintained at the FITCPAC facility to facilitate the 
realistic-scenario driven simulations conducted by the military instructors without 
disturbance to the real world. 

Scenarios are put together by contractors with experience in scripting, doctrine, 
and various applications utilized in order to create realistic events. The process of 
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creating these scenarios is very tedious and takes two qualified contractors approximately 
a year to build. First, research into doctrine of the countries involved must be conducted 
so as to script rational results. Secondly, orders of battle must be loaded in from the 
stand-alone MIDB that include all countries involved which are placed in specific areas 
according to the research conducted by the contractors. Thirdly, event-driven scenarios 
are scripted one enemy reaction at a time dependent on the action of the user. 

The current trainer progresses through the interaction of the watch standers and 
instructors based on the decisions made by the watch standers using data provided for 
them. The watch standers interact through chat and analyzing message traffic and visual 
data provided for them by display. The watch standers produce a complete intelligence 
picture for the Force Intelligence Watch Officer (FIWO), who in turn makes 
recommendations to “the staff’, which is simulated by the FITCPAC instructors who are 
observing the scenario. The team’s recommendations are always taken by the staff and 
the system runs its scripted actions. 

D. SYSTEM DEFICITS 

The current system’s forces initiate actions as a result of scripted programming. 
In other words, forces will execute the same actions regardless of what the user decides to 
do, resulting in highly unrealistic scenarios. For instance, in a scenario where an ARG 
transits the Strait of Hormuz, the response of Iranian P3 or an incoming dhow would be 
the same whether the battle group utilized frigate escorts or not. Because of this, scripted 
events are generally inferior to AI for the overall objective of FITCPAC’s Intelligence 
Team Trainer. 

Due to the amount of time it takes to create a complete scenario, the FITCPAC 
facility will have only two or three scenarios available to them per training team cycle. 
This results in the intelligence team running each scenario approximately three times, 
each time with same situation and same behavior of enemy forces. Scripted events limits 
the amount of reaction an enemy force will have as a result of the user’s action. The 
team can become lethargic, simply because if they are fa mi liar with the scenario, they can 
repeat previous actions with complete knowledge of what the enemy will do. The 
training becomes monotonous, and users will continue on with the scenario without 
reaching their full strategic potential. 
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E. POSSIBLE SYSTEM IMPROVEMENTS 

The best way to improve this training is to use AI to create enemy forces (often 
referred to as “red forces”) and U.S. forces (“blue forces”) which will react to one 
another. Therefore, each time the same simulation set-up is rendered, the behavior of the 
enemy forces will not necessarily be the same, since the actions the trainees take will 
change the behavior of the blue forces which will effect the behavior of the red forces. 
Furthermore, it is possible to introduce variation/randomness in the behavior of the 
enemy agents even given identical actions by the user. Adding AI like to the current 
system will improve the realism of scenarios for the training facility at FITCPAC. 
Additionally, such a system will be deployable with each shipboard intelligence team. 
Building a system to introduce such variability in force behavior is the aspect we focus 
on in this work. 

Another advantage of utilizing AI within the current system is that it will 
significantly reduce the time spent on creating scenarios. As mentioned earlier, it takes 
approximately a year to generate a scripted scenario. Adding pull-down menus to 
establish each scenario prior to running it will significantly shorten time spent on 
creation. Of course, creating a system that enables a scenario to be drafted regardless of 
what country and area of responsibility will require a significant amount of research. 
This thesis covers only one scenario, involving the ongoing tensions with the arms race 
of China and its presumed desire of Taiwan via a situation involving the Straight of 
Taiwan. Details will be discussed later in Chapter IV. 

Additionally, by generating message traffic with the AI the instructors will have 
the opportunity to be continuously present in the trainer with the team, vice at modules 
typing up responses. This will produce a better training environment because the 
instructors will have more time to critique and mentor the students. The system will also 
have the capability to record, stop, and replay a scenario at the instructors’ preference. 
This functionality will allow for the optimum training environment for the instructors to 
discuss with the team their decisions and how to improve their decision making process. 
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IV. SOFTWARE PROTOTYPE 


A. INTRODUCTION 

This chapter will provide an overview of the software developed to support the 
experimental investigation described later. 

B. SYSTEM ARCHITECTURE 

The game-based simulator was created using NetBeans, which is an open source 
integrated development environment (IDE) written entirely in Java using the NetBeans 
Platform. NetBeans is based upon the Java language which contains the ability to 
provide simple graphical user interface (GUI), making it easier to develop complex 
behavior. Java contains modules to handle networking, rendering, movement, and 
complex interactions between objects. The Java code will act like a game engine as 
shown in the architecture diagram. Figure 4. 



Figure 4. Architecture Diagram 


The program was written utilizing various classes that control different aspects of 
the game-based simulator. Each class has a specific purpose within the simulation such 
as setup of initial variables, generating message traffic, movement of forces, and 
generation of the GUI. 
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The initial architecture designed for this thesis is limited to just the scope of the 
thesis. The application demonstrates for the user the distinction between a fixed-result 
simulation and a simulation developed with AT 

C. FEATURES 

In order to create a miniaturized stand-alone version of the current system, steps 
were taken so that all applications utilized in this test platform are as similar as possible 
to the real system. Miniaturized versions of systems include: the GUI, MIDB, Message 
Traffic Generation, and an addition to the current system which is the pull-down scenario 
creator ( See Figure 5). 

1. Graphical User Interface 

All the features involved within the simulation are conducted in a GUI built 
within the NetBeans IDE. The opening display allows the user to setup the simulation 
from a series of buttons that open up separate windows with allow users to choose 
different preferences on how they would like to run the simulation. This allows the user 
to select area of operation (AO), time of day, and intensity level. Each selection 
determines the variables involved within each class and function that help govern the 
course of the simulation. A brief description of each class and its purpose within the 
simulation is discussed later in this chapter under Implementation. 
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Figure 5. Screen Shot 1 of Simulation 


2. Modernized Integrated Database (MIDB) Structure 

First, research was conducted in order to analyze China’s military doctrine and 
history of reactions to various combat situations. Additionally, Taiwan was studied in 
order to establish its military standing and social dependency on China. A very simple 
order of battle was established for China, Taiwan, Britain, Australia, Japan, and the 
United States. Assumptions of course were made that all parties would be involved on 
the influence of the United Nations and previous experience with established coalitions 
during Operation Enduring Freedom. The original MIDB would provide a substantial 
amount of intelligence data, but for the purposes of this thesis, data is limited to only data 
needed to conduct this experiment. 

The MIDB be used in this thesis will be based on a plain text file built via the 
very simple and well known text editor Notepad. In future versions, Microsoft Excel or 
Intel Excel should be used to simplify the entry process. Notepad allows for simple 
extraction of the data needed to establish order of battle for red and blue forces. The data 
extracted will include the following: 
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• Military bases and facilities (i.e. port, airbases, SAM sites) 

• Military units (i.e. types of ships, aircrafts) 

• Weapons associated with each platform 

• WMD 

3. Message Traffic Generation 

Intelligence information gathered from and transmitted via message traffic is 
crucial for the BG intelligence teams. Messages received via the Naval Computer and 
Telecommunications System (NCTS) are transmitted and disbursed throughout all U.S. 
Naval Ships and Shore Facilities. Information contained within each message must 
follow a specific format encompassing subject matter that is classified as unclassified, 
confidential, secret, or top secret. Format of messages are in accordance with Naval 
Telecommunications Procedure (NTP3(J)97) and for the purposes of this thesis, all 
messages’ will be labeled as confidential; however, no actual confidential information 
will be created or used (Figure 7, Figure 8). 


m 


Q Intelligence Reports 


Messages \o Chat !0 Contacts 


9 Messages 


D 210001Z 


ZNYCCCCC 
P 210001Z MAR 06 

FM NAVbANTFITCPAC NORFOLKVA//N30// 

TO USS SHIP//JJJ// 

INFO CINCU^NTFLT NORFOLK VA//N5// 

BT 

CONFIDENTIAL//NO2300// 

NEWS REPORTS THAT TAIWAN ABANDON'S "ONE-CHINA" 
PRINCIPLE BUT DOES NOT DECLARE INDEPENDENCE. 
CHINA REMOVES RIGHTS FOR US NAVAL SHIP PORT 
CALLS. 


Figure 6. Message Traffic Generator Initial Message 
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Q Intelligence Reports 


^ Messages fO Chat \o Contacts 


D 210005Z 
D 210005Z 
D 210006Z 
D 210007Z 
D 210007Z 

D 


210008Z 


D 210009Z 


ZNYCCCCC 
P210008Z MAR 06 

FM NAVLANTFITCPAC NORFOLK VW/N30// 

TO USS SHIP//JJJ// 

INFO CINCLANTFLT NOR FOLK VA//N5// 

BT 

CONFIDENTIAL//NO2300// 

REPORT OF UNKNOWN RADAR CONTACT OFF THE NORTH 
TIP OF TAIWAN AT SPEED 400 KNOTS HEADING TOWARD 
THE CAPITAL OF TAIWAN (TAIPEI). 


Figure 7. Message Traffie Generator 


The purpose of the message traffic generation within the thesis scope is strictly 
confined to demonstration of the established scenario in the thesis simulation. In future 
work, AI should be utilized to generate messages as deemed necessary by user’s actions. 
Messages in this simulation however, are messages that are pre-written and will emerge 
within the simulation at pre-determined time increments. For example, at time x, 
message 02 will appear; at time x +1, message 03 will; appear and so forth. 

D. IMPLEMENTATION 

This section will describe briefly the structure of classes involved in the 
demonstration of this thesis. 

I. Selected Classes and Methods 

The simulation uses the following classes and methods and a concise summary of 
each is shown: 
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Settings class . This class is the initial display which allows the user to choose 
various options available to determine the variables involved in the classes listed below 
to run the simulation. For example, time of day, area of operation, and intensity of the 
scenario. 

Start class . This class runs the GUI to get instructor inputs that determine what 
happens within each time dependent portion of the simulation. Within the GUI there is 
the chart of the area of operation (in this case Strait of Taiwan), controls that stop and 
start the simulation, and the display that allows the user to scroll through message traffic 
available to him/her at various times throughout the demonstration. See Figure 5. 



Figure 8. Screen Shot of simulation 


Coordinate class . This class was written to assist in icon placement and 
movement throughout the 2D environment on the chart. Movement is dependent on the 
speed of the icon (i.e. ship, aircraft, or static shore facility) which is shown by the speed 
vector in the direction of movement. 
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MessageTree class . This class provides the message generation and message 
distribution of the simulation. 

SimElement class . This class is an abstract class which has methods and 
functions that are similar to all aircraft, surface vessels, and submarines, (i.e., speed, 
quantity, location). Classes that implement this interface are those written to control each 
subclass such as AircraftElement, ShipElement, and so forth. This class was written in 
order to ease the complication of creating a list of objects and allowing easy movement of 
each regardless of the entity. 
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V. TEST SCENARIO 


A. INTRODUCTION 

This chapter will provide a brief summary of the history and politics behind the 
chosen scenario of the Strait of Taiwan is given. Additionally, this chapter will 
summarize the initial system demonstration setup and parameters involved in the pilot 
experiment that was conducted in order to demonstrate the importance of exploiting AI 
technology in military training systems. 

I. Background Information on Scenario 

Since the presidential eleetions of March 2000, the world’s attention is once again 
focused on Taiwan. There has been an ongoing and increasing concern over the island’s 
security, in particular, whether not China will use force against Taiwan and of course, 
whether or not the U.S. will intervene to help defend the small island of Taiwan. 

The rapid rise of the Peoples Republic of China (PRC) as a regional political and 
economic power with global aspirations is one of the principle elements of the emergence 
of East Asia. As a result of China’s military expansion and the expressed political desires 
of the concept of “one-China”, the focus of the PRC is clearly the prevention of Taiwan 
independence. (Studeman98) Independence has not been declared by Taiwan, but despite 
negotiations that been ongoing between both parties, Taiwan has made it known to the 
world that they will never settle on Beijing’s current terms. 

The likelihood of a conflict with China as a consequence of force against Taiwan 
is a realistic possibility. Therefore, a scenario based on this exact conflict involving the 
prospect of U.S. intervention with variables that would play influential roles in decision 
making is ideal for this investigation of AI. 

B. INITIAL SYSTEM DEMONSTRATION SETUP AND PARAMETERS 

The generation of the scenario depends upon the selections made in the initial set 
up process which is conducted by the instructor in this experiment. Once the selections 
have been made the instructor is responsible monitoring the simulation and providing the 
keyboard inputs that will drive the AI of the system. The instructor’s involvement will be 
discussed later in this chapter in detail. 
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1. Scenario Generation 

As discussed in Chapter IV, the scenario chosen for this experiment is the 
potential threat of conflict with China if a hostile situation develops between China and 
Taiwan. The territory directly involved within the scenario will be the body of water 
called the Strait of Taiwan, as shown in Figure 9. 


The graphical user interface provided by this simulation starts off with the user 
choosing what region of the world they would like to operate in. The setup lists other 
areas of the world; however, only the Strait of Taiwan will be active in this thesis. In 
conjunction with selecting the region of interest there will be other options as discussed 
in chapter IV, such as, time of day, and intensity level. All of these selections shown in 
the setup will display various options, but again, only the selection needed for the 
scenario will be active. This will allow for future implementation and provide the 
illusion of a complete training system. 
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Figure 9. 





The scenario was conducted in two versions depending on the stage of the 
experiment. The first version was the fixed response version; in other words, no AI 
involved and actions of the enemy will be simulated in pre-determined time increments. 
A story board, shown in Table 1, was created that illustrates the complete scenario in a 
table format that lists the time, event, expected action, and the contingent action of the 
simulation. Again, this version will be time driven and will have no independent 
reactions other than what is scripted. 


Time 

in 

(HH:MM) 

Event 

Expected Action 

Contingent Action 

00:00 

Commence Transit 



00:01 

News Input 

News reports that Taiwan 
abandons “one-China” principle 
but does not declare 
independence. China removes 
rights for US port calls. 

Instructor make note 
of reaction. 

00:02 

Report of Snoop Tray-2 
radar from EP-3 on patrol 
over Strait at location 
south of Taiwan 

Match radar to platform (Kilo) 
Ask for HUMINT, SIGINT, and 
imagery operators for any 
information on Chinese Kilos. 

Instructor make note. 

00:03 


Recommend vectoring P-3 
supporting ESG transit to Kilo 
position for localization 

Kilo will be located 
within its normal 
range at T H-01:00 

00:04 


Watchstander should recognize 
that Chinese Kilos carry shoulder 
launched SAM’s with range of X 
NM and inform watch captain. 
Watchstander should recommend 
to watch captain that P-3 maintain 
safe standoff distance via USWC. 

Receive report that 
Snoop Tray-2 Ceases 
Emitting and receive 
report of Chinese 
submarine exercise 
ongoing. 

00:05 

Report of local Taiwanese 
fishing vessels being 
harassed off coast of 
Taiwan by Chinese 
frigates 

Watchstander should be 
constantly monitoring all contacts 
on display. If unknown contacts, 
should be contacting other ships 
near contact’s location for contact 
information. 

Instructor make note. 

00:06 


Watchstander should locate 
Chinese frigates on display; if 
not, trying to correlate with 
contacts on display by contacting 
other units in area for their 
INTEL. 

Chinese Erigate will 
be shown changing 
course in opposite 
direction and 
becoming a non threat. 

00:07 

Unknown surface contact 
heading import. 

Watchstander should be interested 
in surface contact and 
identification. 

P-3 will report that it 
is a Commercial 
Tanker heading in 
port, (will longer be an 
issue) 
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Time 

in 

(HH:MM) 

Event 

Expected Action 

Contingent Action 

00:08 

Unknown radar contact 
off the North tip of 
Taiwan at speed of X 
knots heading toward the 
capital of Taiwan 
(Taipei). 

Watchstander should have noticed 
the unknown contact was an 
aircraft from speed and already 
made attempt to identify through 
means available to him/her. 

CG will report that it 
shows IFF of a 
Chinese J-6 and they 
are conducting a 
query. 

00:09 


Watchstander should report 
weapons capability of J-6 and 
recommend to watch captain to 
label contact as hostile. 

CG will report that 
query of aircraft was 
ignored and they are 
heading in direction of 
CG off north coast of 
Taiwan. 



Watchstander should recommend 
diverting CAP in direction of 
aircraft for a visual and request of 
intentions and change of course. 

Instructor make note. 

00:10 



J-6 will continue to get 
closer to CG and emits 
fire control radar. 

00:10 

Scenario Ends. 




Table 1. Story board for scripted version 


The second version is supported by AI and the simulation will progress with the 
keyboard inputs of the instructor dependent on the response of the user. Similar to the 
fixed-version, there is a story board, shown in Table 2; however, the story board involved 
with the AI version tells the instructor with what key to push (1, 2, or 3) during certain 
time blocks in response to the user’s actions. The total time for the AI version will be the 
same as the fixed-version, 10 minutes. 


Time 
in (HH: 
MM) 

Event 

Expected Action 

Key 

Contingent Action 

00:00 

Commence 

Transit 




00:01 

News input 

News reports that Taiwan 
abandon’s “one-China” 
principle but does not declare 
independence. China removes 
rights for US port calls. 

1 

If threat to level of transit is 
noted, scenario will continue as 
normal. (Note for instructor) 
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Time 
in (HH: 
MM) 

Event 

Expected Action 

Key 

Contingent Action 




2 

Instructor dependent: Moder,..” 
Le--^1 of prefereiu : 




3 

If this is not noted by the watch 
standers as an increased threat 
level of transit, then scenario 
should become more difficult. 

(Note for instructor) 

00:02 

Report of 
Snoop Tray-2 
Radar from 
EP-3 on patrol 
over Strait at 
location south 
of Taiwan 

Match radar to platform (Kilo). 
Ask for HUMINT, SIGINT, 
and imagery operators for any 
information on Chinese Kilos. 



00:03 


Recommend vectoring P-3 
supporting ESG transit to Kilo 
position for localization 

1 

If instant reaction is: P-3 is 
vectored. Kilo will be located 
within its normal range at T 
H-01:00 




2 

Kilo will be located at T 
h-02. )0. 




3 

If P-3 is not vectored. Kilo will 
disappear from radar and 
location will be unknown 
leaving ESG vulnerable. 

00:04 


Watchstander should recognize 
that Chinese Kilos carry 
shoulder launched SAM’s with 
range of X NM and inform 
watch captain. Watchstander 
should recommend to watch 
captain that P-3 maintain safe 
standoff distance via USWC. 

1 

If so, receive report that Snoop 
Tray-2 ceases emitting and 
receive report of Chinese 
submarine exercise ongoing. 




2 

Receive report that Chinese 
"ubmarine exercise ceased and 
hrr intended mo. -nijnt 
eward ESG 




3 

If not, P-3 might be engaged by 
shoulder launched SAM and 
receive HUMINT relay report 
of Kilo-366 being loaded with 
torpedoes 6 days ago before 
departing homeport. 

00:05 

Report of local 
Taiwanese 
fishing vessels 
being harassed 
off coast of 
Taiwan by 
Chinese 
frigates 

Watchstander should be 
constantly monitoring all 
contacts on display. If 
unknown contacts, should be 
contacting other ships near 
contact for contact information. 

1 

If this happens, Chinese frigate 
will be shown changing course 
in opposite direction and 
becoming a non threat. 
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Time 
in (HH: 
MM) 

Event 

Expected Action 

Key 

Contingent Action 




2 

Continued harassmen: of 
fishing vessel will lead to 
bridge-to-bridge transmission 
of SOS. And frigate will cease 
hara^smeni. 



Watchstander should locate 
Chinese frigates on display, if 
not, trying to correlate with 
contacts on display by 
contacting other units in area 
for their INTEL. 

3 

If this does not happen, Chinese 
frigate will continue to harass 
and Taiwanese response will 
elevate and could lead to shots 
fired toward fishing vessel. 

00:06 

Unknown 

Surface 

contact 

heading 

import. 

Watchstander should be 
interested in surface contact 
and identification. 

1 

If user asks about the unknown 
surface contact, P-3 will report 
that it is a commercial tanker 
heading in port, (will no longer 
be an issue) 




2 

Message that Intel received on 
certain shipping vessels 
carrying biological actents 
onboard and .e kecD leak ..;t. 




3 

If not, receive report of 
mobilization of PLA in 
preparation for attack of 
Taiwan. 

00:07 

Unknown 
radar contact 
off the North 
tip of Taiwan 
at speed of X 
knots heading 
toward the 
capital of 
Taiwan 
(Taipei). 

Watchstander should have 
noticed the unknown contact 
was an aircraft from speed and 
already made attempt to 
identify through means 
available to him/her. 

1 

If so, CG will report that it 
shows IFF of a Chinese J-6 and 
query in progress. 




2 

If noticed and did nothing to 
find out, CG will report that 
query being conducted and 
r e'onsL )ng')in,e 




3 

If not, CG will report that query 
of aircraft was ignored and they 
are heading in direction of CG 
off north coast of Taiwan. 



Watchstander should report 
weapons capability of J-6 and 
recommend to the watch 
captain to label contact as 
hostile. 

1 

CG will report query successful 
and J-6 changing course. 




2 

CG will conduct a 2"'* Query 
and warn J-6 that they will 
shoot to kill. 




3 

J-6 will ignore query and 
continue heading towards CG. 
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Time 
in (HH: 
MM) 

Event 

Expected Action 

Key 

Contingent Action 



Watchstander should 
recommend diverting CAP in 
direction of aircraft for a visual 
and request of intentions and 
change of course. 

1 

If so, J-6 will change directions 
and head back in the opposite 
direction. 




2 

If suggest something else, 1-6 
will come within 10 nm and 
turn around. CG will report 
thi:- 




3 

If not, J-6 will continue to get 
closer to CG and emits fire 
control radar. 

00:10 

Scenario Ends. 





Table 2. Story board for AI version 
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VI. EXPERIMENT 


A. EXPERIMENTAL SETUP TO VALIDATE THE USE OF THE SYSTEM 

This chapter will describe the experimental design and subjects used to conduct 
the experiment. 

1. Subjects Used to Conduct Experiment 

The experiment was conducted using students qualified in Naval Surface Warfare 
from the Naval Postgraduate School in Monterey, California. It is essential that the 
subjects have the basic knowledge and understanding of how the Global Command and 
Control (GCCS) display functions, experience in dealing with contacts of various entities 
and the threat of proximity in international waters, as well as rules of engagement (ROE) 
in a potential combat zone. 

This experience comes from standing watch while underway onboard a combat 
capable ship and obtaining either the Combat Information Center Watch Officer 
(CICWO) or Tactical Actions Officer (TAO) qualification. The requirement for the 
qualification in Surface Warfare for this experiment fulfills the basic knowledge 
constraint, because CICWO is a prerequisite to be Surface Warfare qualified. 

2. Experimental Setup 

The experiments were conducted at the Naval Postgraduate School in which 
student volunteers, qualified in Surface Warfare, were used to determine which of the 
two systems used in this experiment were superior. First, the subjects were informed that 
the training simulation was a simple reconstruction of a current system used at FTCPAC 
that included the basic functionalities and are limited to one scenario each for 
experimental purposes. Additionally, they were instructed in how to conduct various 
tasks within the GUI and what they were expected to accomplish within the training 
simulation. 

The experiment was conducted in two groups: 1) control group and 2) 
experimental group. The control group was made up of subjects who operated the fixed- 
version training simulation and the experimental group subjects were operating the AI- 
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version training simulation. Subjects were randomly assigned to the two groups by 
choosing every other one based on their arrival time. 

Control group : conducted a total of three simulation runs of the fixed-version 
simulation. 

Experimental group : conducted a total of three simulation runs of the AI-version 
simulation. 

At the end of the test runs, there was a face-to-face debrief, which included a 
questionnaire for each group that was used afterward for comparison analysis. The 
questionnaire addressed questions for each event of the simulation, separately, in order to 
determine how many predictions a subject anticipated for the enemy’s reactions within 
that particular event. (See Appendix A for the questionnaire developed for this 
experiment) 

3. Experimental Design 

The purpose of this experiment is to demonstrate the importance of applying AI 
technology to current training simulations. The system being enhanced currently uses 
scripted events that carry out enemy actions independent of the user’s inputs/reactions. 
This is extremely detrimental to overall training because it does not take into account the 
variability of the real world. After the first run of a particular scenario, every run 
thereafter is one-hundred percent predictable because the enemy will perform the same 
actions every time. In other words, the subject is expected to predict only one enemy 
response per event with the scripted version. Ironically, the current system does produce 
overall performance improvement between scenario runs; however, this is only because 
the user knows exactly what is going to happen. 

The use of AI technology not only will keep the user engaged, but enhance 
overall combat readiness (Boldo87). This is of course important to any military branch, 
for the threats in the world today are much different than those a generation prior. The 
military today is dealing with threats that are non-conventional and unpredictable in a 
sense that we are operating in environments that are not familiar (e.g., urban warfare or 
religious persecution). 
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The subjects were not told what system they will be assigned to, only that they are 
either in system A or system B. A scripted format of steps to be taken was strictly 
followed in order to ensure consistency throughout the experiment. (See Appendix B) 

The variable to be analyzed was the number of predictions for the enemy’s 
response within each event, indicated by the user in the questionnaire. 

The hypothesis of the experiment is: Adding AI to a training simulation causes 
the subject to consider more possible enemy responses per event throughout multiple runs 
of the simulation. 

B. ANALYSIS 

Data obtained from this experiment is the number of enemy response predictions 
per event. Each subject answered a questionnaire which lists each event separately 
requiring a number that describes how many of the enemy’s responses they anticipated 
during that event given the history of the last three runs of the simulation. In other 
words, if the subject ran a fourth experiment, what they would expect to be the enemy’s 
response based off of what had happened in the last three simulation runs. 

The control group responded as expected given the exact same scenario three 
times in a row. Because the enemy responded the same all three runs, they anticipated 
that the fourth run would be the same. The results are shown in Table 3. 
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Table 3. Distribution of Average Number of Anticipated Responses for Control Group 
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C. RESULTS OF EXPERIMENT 

Is hypothesis correct? Adding AI to a training simulation caused the subject(s) to 
consider more possible enemy responses per event on the final “Test” run (See Tables 4, 
5,6). 

AI Version 


□ Average 


1 2 3 4 5 6 7 

Events 


Table 4. Distribution of Average Number of Anticipated Responses for Experimental 

Group 



Fixed Version 

AI version 

Event 

Average Number of 
anticipated Enemy 
response(s) 

Average Number of 
anticipated Enemy 
response(s) 

Event 1 

1.0 

2.5 

Event 2 

1.0 

3.75 

Event 3 

1.0 

3.75 

Event 4 

1.0 

3 

Event 5 

1.0 

3.25 

Event 6 

1.0 

2.5 

Event 7 

1.0 

2.5 


Table 5. Table of Data Obtained from Experiment 
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Distribution of Anticipated Enemy Responses 


1 2 3 4 5 6 7 

Event 

Table 6. Distribution of Anticipated Enemy Responses, Displaying Control Group vs. 

Experimental Group 
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However, while we did not perform a statistical test, the data makes clear that the 
null hypothesis (that the responses of control and experimental groups are from the same 
distribution) must be rejected, since there is almost no overlap between the two response 
sets. The control group uniformly responded with “1” for the number of possible enemy 
responses. In the experimental group, only one individual in one instance gave the 
response “1”. See complete data from the experiment in Appendix C. 

1. Analysis of Results 

Common sense tells us that the results of conducting an experiment with this 
setup will result in more predictions of anticipated enemy responses in the AI version 
versus the fixed version training system. However, this is dependent on the setup and 
consistency in which the experiment is presented to the subjects. 

This approach to enhancing legacy military training systems by incorporating AI 
into the already existing system is very simple and will ensure that the subjects are being 
presented with an environment that is more realistic. 

When subjects use a training tool whose objective is to increase combat readiness, 
how combat ready is a person whose only reference to combat is a training tool that is 
one hundred percent predictable? 
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2. Control Group 

All subjects who were in the control group realized that the enemy responses were 
fixed during the second run of the simulation. Of the four subjects who conducted the 
controlled experiment, the first three verbally noted that ""the enemy did the same thing 
last time”, when the third event occurred in the second simulation run. While the fourth 
subject realized the enemy had the same reaction during the fourth event of the second 
simulation run. Upon realizing that the enemy’s actions were going to be the same 
regardless of what they did, the subjects were not as observant as in the beginning and all 
pursued some sort of conversation outside the area of the experiment. 

Results of the control group confirmed the complacency assumption of scripted 
events. Basic human nature applied and subjects were not as observant of the scenario 
and its components when they came to understand that they did not control or effect the 
enemy’s reactions. 

3. Experimental Group 

Contrary to the control group, the subjects in the experimental group understood 
that their actions (verbally) decided the actions of the enemy. Two subjects noted that the 
key strokes made during each event made them believe that their responses were being 
addressed and that the enemy was reacting because of the key strokes. Whether or not 
this was the case, the key strokes played a essential role in keeping the subjects observant 
during the simulation runs. 

During the second run of the simulation all subjects understood that the enemy 
was reacting differently than the first simulation run. This was mainly due to the fact that 
during the first simulation run, every subject had a high intensity reaction from the 
enemy. This increased intensity level was set because each subject was getting use to the 
controls and functionality of the simulation during the first run and most tended not to be 
as verbal as needed by the experiment. Once each subject was settled into their own 
routine the second and third run went more smoothly with less and less intense reactions 
from the enemy. 
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4. Other Results 

Overall the simulation runs of eaeh group went smoothly and as pereeived. 
However, one subjeet did not understand what was needed from him in the debrief 
questionnaire and because too much information was given to him, the answers received 
by this subject had to be thrown out. 
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VII. SUMMARY, CONCLUSIONS, AND FUTURE WORK 


A. SUMMARY 

In the real-world, the enemy will not always react in the same way when 
presented with an identical situation or environment. So why should we be training our 
military personnel on a system that uses this antiquated approach. 

Similar to the ‘Choose your own adventure’ books, a series of children’s books 
first published by Bantam Books from 1979 to 1998, the framework of this system 
requires numerous potential responses in the database. This will ensure that the user’s 
actions will result in a different enemy action each time the simulation is played at least 
more than four runs consecutively. The number of available enemy responses will be 
dependent on the system design and intention of training. 

In a paper written by Francis Kapper in 1981, he summarizes proper use of 
military simulations and the potential level of validity achievable by those simulations: 
“The most appropriate and valid objectives for using war games and simulations within 
the DoD context are to: better understand complex phenomena, identify problems, 
evaluate alternatives, gain new insights, and broaden one’s perspectives” (Kapp81). The 
variability of the enemy’s responses to the user’s actions was accomplished with a very 
simple technique allowing the enemy to have more than one response. The three to four 
responses available to choose from dependent on the users actions was enough to ensure 
the subject was engaged and maintained continuous observation of his/her simulation 
environment. As mentioned earlier, scripted game-based environments tend to lose the 
attention of users once they realize no matter what they do, the enemy will always react 
the same. 

B. CONCLUSIONS 

In recent years there has been a dramatic interest in developing simulations and 
virtual environments for training purposes, particularly in the military. The demands on 
military simulations are continually increasing, primarily motivated by fiscal necessity 
and secondly, due to the pressure to employ such simulations, which has led to 
exploration of new methods for modeling combat activities. Billons of dollars are spent 
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producing simulations for combat training; unfortunately, little is spent on validating the 
products and even fewer spent on the effects of utilizing artificial intelligence (AI) to 
enhance legacy military training systems. 

The purpose of conducting this experiment in such a manner as described above is 
to demonstrate the importance of applying AI technology to current combat-related 
training systems, despite the simplicity of the AI. This work demonstrates that simple AI 
is better than no AI. 

Again, most legacy combat-related training tools use scripted events that carry out 
enemy actions dependent of the user’s inputs/actions. This is extremely detrimental to 
overall training because it does not take into account the variability of the real world. 
After the first run of a particular scenario in a scripted system, every run thereafter is one- 
hundred percent predictable because the enemy will react the same way every time. In 
other words, the subject is expected to predict only one enemy response per event with 
the scripted version. Ironically, these systems will produce overall, performance growth; 
however, this is only because the user knows exactly what the enemy will do. 

C. FUTURE WORK 

A recommended extension of the experiment would be to execute a total of four, 
simulation runs with subjects in both the control and experimental group. The fourth 
simulation run would be carried out with a completely different scenario utilizing AI for 
both groups. The reaction time of each user could be recorded to determine performance 
levels based on the experience obtained during the first three runs within their own group. 
This could establish the application of AI in determining the efficiency of training when 
presented with a completely new scenario. 

Further research and development of techniques in application of AI in combat 
training tools is highly encouraged, for the reason that it can greatly contribute to the 
combat fighting proficiency of our future Sailors. In particular, in order create a more 
useful training tool for military intelligence team trainers; AI for chat capabilities with 
other entities in the Battle Group should be developed. 
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APPENDIX A. DEBRIEF QUESTIONNAIRE 


This handout was used in order to obtain the data needed for the thesis hypothesis. 
The subject was told to answer the questions with respect to only the simulation and their 
experience with the three simulation runs. In other words, if the subject ran a fourth 
experiment, what they would expect to be the enemy’s response based off of what had 
happened in the last three simulation runs. 



Patricia A. Sweat 
MOVES 
Watkins Hall 

Naval Postgraduate School 
Monterey, California 93943 


831-583-0988 
541-771-2782 
pasweat @ nps.edu 


DEBRIEF QUESTIONNAIRE 

First Event : (Report of Snoop Tray-2 Radar from EP-3 on patrol over Strait at 
location south of Taiwan) 

How many reactions of the enemy did you anticipate given the current situation? 

I I Once P-3 vectored. Kilo located instantly 
I I Kilo’s exact location could not be determined. 

I I Kilo not located and datum disappears from radar, leaving BG and ESG 
vulnerable. 

QEist other reactions::_ 


Second Event : (Chinese Kilo threat determination) 

How many reactions of the enemy did you anticipate given the current situation? 
I I Receive report of Chinese submarine exercises ongoing. 


43 







I I Receive report of Chinese submarine intended movement towards area of 
Battle group or ESG. 

I I P-3 might be engaged by shoulder launched SAM 

I I Receive report of Kilo-366 being loaded with torpedoes prior to departing 
homeport. 

I I List other reactions:_ 

Third Event : (Report of Local Taiwanese Lishing vessels being harassed off 
coast of Taiwan from Chinese Lrigates) 

How many reactions of the enemy did you anticipate given the current situation? 

I I Chinese Lrigate change course and stop harassment. 

I I Continued harassment from Chinese. 

I I Call for help from fishing vessels over Bridge-to-Bridge. 

I I Continued harassment of fishing vessels lead to elevation to Taiwanese 
response. 

I I List other reactions:_ 

Fourth Event : (Unknown Surface contact heading import) 

How many reactions of the enemy did you anticipate given the current situation? 

I I Identified as Chinese Naval Vessels. 

I I Receive report that it’s a commercial shipping vessel. 

I I Receive report of commercial vessels carrying biological agents onboard. 

I I No response from surface vessel leading to concern of threat to Taiwan. 

I I List other reactions:_ 
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Fifth Event : (Unknown radar contact off the North Tip of Taiwan at speed of X 
knots heading toward the capital of Taiwan - Taipei). 

How many reaetions of the enemy did you antieipate given the eurrent situation? 

I I Report of IFF and eontact identified. 

I I Report that eontact Chinese aircraft and query in progress. 

I I Report that query is in progress with eontaet and no response. 

I I Air contact heading towards ESG. 

I I List other reaetions:_ 

Sixth Event : (Query of Air eontact in progress) 

How many reaetions of the enemy did you antieipate given the eurrent situation? 

I I Report that query successful and aireraft ehanging eourse. 

I I Report first query suecessful however, aireraft not ehanging eourse and 
seeond query ongoing questioning intentions. 

I I Report that queries ignored and preparing for Battle Stations. 

I I List other reaetions:_ 

Seventh Event : (Air eraft movement in progress) 

How many reactions of the enemy did you antieipate given the eurrent situation? 

I I Aircraft will change course and head in opposite direction. 

I I Aircraft will continue on eourse towards Battle Group getting elose enough to 
upset, but turn around in time not to be shot at. 

I I Aircraft will continue on course towards Battle Group and emits fire eontrol 
radar. 
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I I List other reactions:_ 

During any time in the experiment did you believe that there was a sense of 
variability with the enemy’s actions? (i.e. did the enemy react in response to your 
actions, or did the enemy react regardless of what you chose to do or say). Briefly 
explain your answer. 

I I Yes 
□ No 

Any comments or recommendations. 


Thank you for your time, have a great day! 
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APPENDIX B. EXPERIMENTAL PROCEDURE 


Note: The quoted paragraphs are what was said verbatim in order to maintain 
eonsisteney throughout the experiment. 

“I will be reading from a seript in order to maintain eonsisteney among the 
partieipants and to ensure that no detail is left out whieh might be devastating to the 
overall results of this experiment.” 

1. Subject will walk in and sit down. 

a. Subject will be given the Participant Consent Form and Minimal Risk 
Statement for signature 

“First, I will be giving you a Participant Consent Form & Minimal Risk Statement 
for signature and for IRB and my records. The purpose of this experimental study will be 
used to improve the use of advanced techniques to enhance future military training 
simulations. Certain details will be left out in order not to affect the experiment, 
however, please feel free to ask after the experiment is completed. There will be three 
phases in this experiment: 1) this 5 minute brief which will be used to familiarize you 
with the training system 2) training and execution phases lasting approximately 5 minutes 
per simulation, during which you will be expected to accomplish a number of tasks 
related to identifying, interpreting data, and making recommendations based on data 
collected, and 3) 5 minute debrief which will consist of a questionnaire.” 

Second, I need to ensure that you understand that this project does not involve any 
reasonably foreseeable risks or hazards greater than those encountered in everyday life. 

Additionally, I must tell you that all recodes of this study will be kept confidential 
and that your privacy will be safeguarded. No information will be publicly accessible 
which could identify you as a participant, and you will be identified only as a code 
number on all research forms. 

All records of your participation will be maintained by NFS for five years, after 
which they will be destroyed. Your participation is strictly voluntary, and if you agree to 
participate, you are free to withdraw at any time without prejudice.” 
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b. Subject will sign consent form. 

2. Subject will be briefed on what the current trainer at FITCPAC accomplishes. 

a. Trainer for Carrier BG Intelligence cells. 

b. Assists FITCPAC instructors in analysis of individual interpretations 
of intelligence information. 

3. Subject will be introduced to the functionality of the simulation provided by 
the GUI. 

a. Subject will be given visual aid document with useful symbology for 
reference through out the simulation runs. 

b. What each symbol represents (i.e. aircraft, surface, sub-surface, 
unknown, friendly, hostile. 

(Symbology Sheet) - “This sheet should be used as a reference for understanding 
the symbology used in this experiment. These symbols are similar to those seen 
in a Combat Information Center on ACDS screen and/or on GCCS screen. Feel 
free to refer to this sheet throughout the experiment.” 

c. How to use the message traffic box 

(Screen Shot Sheet) - “This is a screen shot of what the simulation will look l ik e 
during the experiment. The tabbed marked ‘Messages’ will be where you check 
messages that are generated that contain reports from various Intelligence sources. 
In order to read the content of the messages you need to click once on each line 
item that shows up at specific times throughout the simulation. 

It’s important to constantly check if new messages have arrived. Messages 
contain information on correlation of contacts and intelligence that will help you 
decided what to do next.” 

d. How to identify various entities on the chart provided by the GUI, (i.e. 
the GCCS display portion) 

(Screen Shot Sheet) - “This is a screen shot of what is the initial contact 

information. As you can see there are some contacts that are labeled ‘unknown’. 
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These are the contacts that you need to correlate with by reading the message 
traffic and analyzing the reports from the Intelligence sources.” 

4. Subject will be told what is expected of them and how to accomplish the 
tasks; via voice or the GUI. 

“During the experiment, you will be expected to verbalize your thoughts, i.e. 
verbalize your explanation for your decision-making. What you wondering about, 
what you think certain entities are or could be, and concerns with certain entities 
and there location in regards to the Battle Group or ESG.” 

a. Information gathering techniques (i.e. messages, voice comms) 

b. Interpretation of data and how to make recommendations to the watch 
captain 

c. How voicing opinion and recommendations is important to the 
progression of the simulation 

5. Once subject understands the tasks involved, the simulation will start. 

a. Subject will conduct four simulation runs overall 

b. Each run will be approximately 5 minutes each 

6. Once all runs of the simulation is over, debrief will commence. 

a. Subject will be given questionnaire. 

b. Explain the questionnaire to the subject and how accurate and correct 
answers are important to the analysis. 

“When answering this questionnaire, please keep in mind that the answers should 
be directing related to the simulation, in other words, what the simulation 
provided to you as a user. When asked how many reactions of the enemy did you 
anticipate given the current event, the answer should be what the simulation 
allowed you to anticipate.” 
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APPENDIX C. EXPERIMENT RESULTS EOR SCRIPTED 

VERSION 


Events 

1 

2 

3 

4 

5 

6 

7 

Subject 1 

1 

1 

1 

1 

1 

1 

1 

Subject 2 
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1 

1 

1 

1 

1 

1 

Subject 3 

1 

1 

1 

1 

1 

1 

1 

Subject 4 
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1 

1 

1 

1 

1 

1 

Average 

1 

1 

1 

1 

1 

1 

1 


Fixed Version 



□ Average 


Events 
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APPENDIX D. CONTACT SYMBOLOGY HANDOUT 


This handout was used as a reference for symbology used in the simulation. Even 
though all subjects were students at Naval Postgraduate School with Combat Information 
Center experience; certain details are forgotten after being away from the environment 
for a long period of time. 
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APPENDIX E. SCREEN SHOT SIMULATION HANDOUT 


This handout was used in order to illustrate what the simulation would look like 
as well as explain the functionality of the message traffic and how to access the 
messages’ content by clicking on the icon. 
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APPENDIX F. SCREEN SHOT MESSAGE TRAFFIC HANDOUT 


This handout was used to illustrate the initial contact identification correlated with 
icons shown on the chart. If for some reason the user forgot what certain contacts were, 
the tab marked ‘Contacts’ was what needed to be clicked to refresh her/his 
memory. 
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APPENDIX G. EXPERIMENTAL RESULTS EOR SCRIPTED 


Events 
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APPENDIX H. EXPERIMENTAL RESULTS 
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APPENDIX I. SCREEN SHOTS OE SIMULATION 
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